Chapter 2 

Matrices and Linear Algebra 



A := uppercase denotes a matrix 
a := lower case denotes an entry of a matrix a € F. 

Special matrices 
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(1) If m = n, the matrix is called square. In this case we have 

(la) A matrix A is said to be diagonal if 

a ij — 0 i / j ■ 

(lb) A diagonal matrix A may be denoted by diag(di, d, 2 , ■ ■ ■ ,d n ) 
where 

an — d/ CLij — 0 j ^ i. 

The diagonal matrix diag(l, 1,... , 1) is called the identity matrix 
and is usually denoted by 

'1 0 ... 0 
0 1 
In= . 

0 1 

or simply I, when n is assumed to be known. 0 = diag(0,... ,0) 
is called the zero matrix. 

(lc) A square matrix L is said to be lower triangular if 

1 ij — 0 i j. 

(l d) A square matrix U is said to be upper triangular if 

Uij = 0 i > j. 

(le) A square matrix A is called symmetric if 

&ij — Qji • 

(l f) A square matrix A is called Hermitian if 

a ij = aji (z := complex conjugate of z). 

(lg) Eij has a 1 in the (i, j) position and zeros in all other positions. 

(2) A rectangular matrix A is called nonnegative if 

> 0 all i,j. 

It is called positive if 

> 0 all i,j. 

Each of these matrices has some special properties, which we will study 
during this course. 
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Definition 2.1.2. The set of all m x n matrices is denoted by M rnAl (F). 
where F is the underlying field (usually R or C). In the case where m = n 
we write M n (F ) to denote the matrices of size n x n. 

Theorem 2.1.1. M m>n is a vector space with basis given by Eij, 1 < i < 
m, 1 < j < n. 

Equality, Addition, Multiplication 

Definition 2.1.3. Two matrices A and B are equal if and only if they have 
the same size and 


O/.j — hjj all i , j. 

Definition 2.1.4. If A is any matrix and a £ F then the scalar multipli¬ 
cation B = aA is defined by 

bij — otdij all %. j. 

Definition 2.1.5. If A and B are matrices of the same size then the sum 
A and B is defined by C = A + B, where 

Cjj — djj T bij all i . j 

We can also compute the difference D = A — B by summing A and (—1 )B 

D = A - B = A + (~1)B. 

matrix subtraction. 

Matrix addition “inherits” many properties from the field F. 

Theorem 2.1.2. If A,B,C £ M rn;n (F) and a, (3 £ F, then 

(1) A + B = B + A commutivity 

(2) A + (B + C) = {A + B) + C associativity 

(3) a(A + B) = a A + aB distributivity of a scalar 

(4) If B = 0 (a matrix of all zeros) then 

A + B = A + 0 = A 


(4) (a + P)A = aA + (3A 
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(5) a((3A ) = a(3A 

(6) OA = 0 

( 7 ) «0 = 0 . 

Definition 2.1.6. If x and y € R n , 

x = («i... x n ) 

y = (yi- ■■y n )■ 

Then the scalar or dot product of x and y is given by 

n 

(®, y) = ^2 XiVi ■ 

i=l 

Remark 2.1.1. (i) Alternate notation for the scalar product: (x, y) = x ■ y. 
(ii) The dot product is defined only for vectors of the same length. 

Example 2.1.1. Let x = (1,0,3,—1) and y = (0,2,—1,2) then (x,y) = 
1(0) + 0(2) + 3(—1) — 1(2) = -5. 

Definition 2.1.7. If A is m x n and B is n x p. Let r,;(A) denote the vector 
with entries given by the z th row of A , and let Cj(B) denote the vector with 
entries given by the j* 11 row of B. The product C = AB is the mxp matrix 
defined by 


Cij = ( ri(A),Cj{B )) 


where ri(A) is the vector in R n consisting of the i th row of A and similarly 
Cj(B ) is the vector formed from the j th column of B. Other notation for 
C = AB 


c 


ij 


n 

T. T/;' bkj 

k= 1 


1 < i < m 
1 <j<P- 


Example 2.1.2. Let 


A 


1 0 1 
3 2 1 


and 



1 

0 

1 


AB 


1 2 
11 4 


Then 
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Properties of matrix multiplication 

(1) If AB exists, does it happen that BA exists and AB = BA? The 
answer is usually no. First AB and BA exist if and only if A G 
M rrh n{F) and B G M n , m (F). Even if this is so the sizes of AB and 
BA are different ( AB is m x m and BA is n x n) unless m = n. 
However even if m = n we may have AB BA. See the examples 
below. They may be different sizes and if they are the same size (i.e. 
A and B are square) the entries may be different 


-4 = [1,2] B = 


A 


1 

3 



1 

1 


AB = [1] 


BA 

AB 

BA 


-1 —2 
1 2 

-1 3' 
—3 7 

2 2 
3 4 


(2) If A is square we define 

A 1 = A, A 2 = AA, A 3 = A 2 A = AAA 
A n = A n ~ 1 A = A - ■ ■ A ( n factors). 


(3) I = diag(l,... ,1). If' A G M m . n (F) then 

AI n = A and 
ImA = A. 


Theorem 2.1.3 (Matrix Multiplication Rules). Assume A,B, and C 
are matrices for which all products below make sense. Then 

(1) A{BC) = ( AB)C 

(2) A(B ±C) = AB±AC and (A ± B)C = AC ± BC 

(3) AI = A and IA = A 

(4) c{AB) = ( cA)B 

(5) AO = 0 and OB = 0 
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(6) For A square 

A r A s = A' s A r for all integers r,s> 1. 

Fact: If AC and BC are equal, it does not follow that A = B. See Exercise 
60. 

Remark 2.1.2. We use an alternate notation for matrix entries. For any 
matrix B denote the (i, j)-entry by ( B)ij. 

Definition 2.1.8. Let A G M m ^ n (F). 

(i) Define the transpose of A, denoted by A T , to be the n x m matrix 
with entries 


(A )ij — Ojj . 

(ii) Define the adjoint of A, denoted by A*, to be the n x m matrix with 
entries 


(A )ij — aji complex conjugate 

Example 2.1.3. 





T 

5' 

'1 2 

3' 

£u 

II 

2 

4 

5 4 

1 


1 




CC 


In words... “The rows of A become the columns of A T , taken in the same 
order.” The following results are easy to prove. 

Theorem 2.1.4 (Laws of transposes). (1) ( A T ) 7 = A and (A*)* = A 

(2) (A±B) t = A 1 ± B t (and for *) 

(3) ( cA) T = cA T ( cA)* = cA* 

(4) ( AB) t = B t A t 

(5) If A is symmetric 


A = A t 
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(6) If A is Hermitian 

A = A*. 

More facts about symmetry. 

Proof. (1) We know ( A T )ij = aji. So (( A T ) T )ij = aij. Thus ( A T ) 1 = A. 
(2) (A ± B) t = aji ± bji. So (A ± B) T = A T ± B T . 

□ 

Proposition 2.1.1. (1) A is symmetric if and only if A T is symmetric. 

(1)* A is Hermitian if and only if A* is Hermitian. 

(2) If A is symmetric, then A 2 is also symmetric. 

(3) If A is symmetric, then A n is also symmetric for all n. 

Definition 2.1.9. A matrix is called skew-symmetric if 

A t = -A. 

Example 2.1.4. The matrix 

"0 1 2 " 

A= -1 0 -3 
-2 3 0 _ 

is skew-symmetric. 

Theorem 2.1.5. (1) If A is skew symmetric, then A is a square matrix 

and an = 0, i = 1,... , n. 

(2) For any matrix A € M n (F) 

A- A t 

is skew-symmetric while A + A T is symmetric. 

(3) Every matrix A € M n (F) can be uniquely written as the sum of a 
skew-symmetric and symmetric matrix. 

Proof. (1) If A € M m>n (F), then A T G M n>m (F). So, if A T = —A we 
must have m = n. Also 

@>ii — G'ii 

for i =s l,... , n. So an = 0 for all i. 
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(2) Since (A — A T ) T = A T — A = —{A — A T ), it follows that A — A T is 
skew-symmetric. 

(3) Let A = B + C be a second such decomposition. Subtraction gives 

l(A + A T )-B = C~l(A-A T ). 

The left matrix is symmetric while the right matrix is skew-symmetric. 
Hence both are the zero matrix. 

A^^A + A^+'-iA-A?). 

□ 


Examples. A— ^ q 1 skew-symmetric. Let 


Then 


B = 
B t = 

b-b t = 
b + b t = 


1 2 
-1 4 

1 - 1 ' 
2 4 

' 0 3' 

-3 0 

2 1 
1 8 ' 


B= l -{B-B T )+ l -{B + B T ). 


An important observation about matrix multiplication is related to ideas 
from vector spaces. Indeed, two very important vector spaces are associated 
with matrices. 


Definition 2.1.10. Let A e M m ^ n (C). 
(i)Denote by 


Cj(A) := j th column of A 

Cj(A ) € C m . We call the subspace of C m spanned by the columns of A the 
column space of A. With c\ (A),..., c n (A) denoting the columns of A 
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the column space is & {c\ ( A),... ,c n (A)). 

(ii) Similarly, we call the subspace of C n spanned by the rows of A the row 
space of A. With n ( A ),..., r m (A) denoting the rows of A the row space 
is therefore 6 (ri (A),... ,r m (A)). 

Let x € C n , which we view as the n x 1 matrix x = \x\ ... x n ] T . The 
product Ax is defined and 

n 

Ax = ^x jCj {A). 

3 =1 

That is to say, Ax € ©(ci(vl),... , c n (A)) = column space of A. 

Definition 2.1.11. Let A G M n (F). The matrix A is said to be invertible 
if there is a matrix B G M n (F) such that 

AB = BA = I. 


In this case B is called the inverse of A, and the notation for the inverse is 
A- 1 . 


Examples. 

(i) Let 


A 


1 3 

-1 2 


A~ l 



Then 


(ii) For n = 3 we have 



1 2 -1 ' 


0 1 -1 ' 

A = 

-1 3 -1 
-2 3 -1 

A- x = 

-1 3 -2 

-3 7 -5 


A square matrix need not have an inverse, as will be discussed in the 
next section. As examples, the two matrices below do not have inverses 


A 



B = 


1 0 
0 2 
1 2 


1 

1 

2 
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2.2 Linear Systems 

The solutions of linear systems is likely the single largest application of ma¬ 
trix theory. Indeed, most reasonable problems of the sciences and economics 
that have the need to solve problems of several variable almost without ex¬ 
ception are reduced to component parts where one of them is the solution 
of a linear system. Of course the entire solution process may have the linear 
system solver as a relatively small component, but an essential one. Even 
the solution of nonlinear problems, especially, employ linear systems to great 
and crucial advantage. 

To be precise, we suppose that the coefficients a- L j , 1 < i < m and 1 < 

j < n and the data bj, 1 < j < m are known. We define the linear system 
for the n unknowns xi, . . ., x n to be 

^ 11 X 1 T ^ 12*^2 T ■ ■ * T ct\ n x n — b\ 

021X1 + 022^2 H-h CL 2n X n = &2 (*) 

OmlXl T CL m 2^2 T ‘ ‘ ‘ T Oj mn X n — b m 

The solution set is defined to be the subset of R n of vectors (xi,..., x n ) that 
satisfy each of the m equations of the system. The question of how to solve 
a linear system includes a vast literature of theoretical and computation 
methods. Certain systems form the model of what to do. In the systems 
below we note that the first one has three highly coupled (interrelated) 
variables. 


3xi — 2x 2 + 4x3 = 7 

Xi — 6 x 2 — 2x3 = 0 

—xi + 3 x 2 + 6 x 3 = —2 

The second system is more tractable because there appears even to the 
untrained eye a clear and direct method of solution. 

3xi — 2x2 — X 3 = 7 
x 2 - 2x 3 = 1 
2 x 3 = —2 

Indeed, we can see right off that x 3 = — 1. Substituting this value into the 
second equation we obtain X 2 = l — 2 = — 1. Substituting both X 2 and x 3 
into the first equation, we obtain 2xi — 2 (—1) — (—1) = 7, gives x\ = 2. The 
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solution set is the vector (2, — 1, — 1). The virtue of the second system is 
that the unknowns can be determined one-by-one, back substituting those 
already found into the next equation until all unknowns are determined. So 
if we can convert the given system of the first kind to one of the second kind, 
we can determine the solution. 

This procedure for solving linear systems is therefore the applications of 
operations to effect the gradual elimination of unknowns from the equations 
until a new system results that can be solved by direct means. The oper¬ 
ations allowed in this process must have precisely one important property: 
They must not change the solution set by either adding to it or subtracting 
from it. There are exactly three such operations needed to reduce any set 
of linear equations so that it can be solved directly. 

(El) Interchange two equations. 

(E2) Multiply any equation by a nonzero constant. 

(E3) Add a multiple of one equation to another. 

This can be summarized in the following theorem 

Theorem 2.2.1. Given the linear system (*). The set of equation opera¬ 
tions El, E2, and E3 on the equations of (*) does not alter the solution set 
of the system (*). 

We leave this result to the exercises. Our main intent is to convert these 
operations into corresponding operations for matrices. Before we do this 
we clarify which linear systems can have a soltution. First, the system can 
be converted to matrix form by setting A equal to the m x n matrix of 
coefficients, b equal to the m x 1 vector of data, and x equal to the n x 1 
vector of unknowns. Then the system (*) can be written as 

Ax = b 

In this way we see that with c* (A) denoting the i th column of A, the system 
is expressible as 


xic i (A) -|-b x n c n (A) = b 


From this equation it is clear that the system has a solution if and only if 
the vector b is in (5 (ci (A), • • • , c n (A)). This is summarized in the following 
theorem. 
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Theorem 2.2.2. A necessary and sufficient condition that Ax = b has a 
solution is that b € 6(ci(A).. ,c n (A)). 

In the general matrix product C = AB, we note that the column space of 
C C column space of A. In the following definition we regard the matrix A 
as a function acting upon vectors in one vector space with range in another 
vector space. This is entirely similar to the domain-range idea of function 
theory. 

Definition 2.2.1. The range of A = {Ax \ x € R n ( orC n )}. 

It follows directly from our discussion above that the range of A equals 
6 (ci(T),... ,Cn(A)). 

Row operations: To solve Ax = b we use a process called Gaussian 
elimination, which is based on row operations. 

Type 1: Interchange two rows. (Notation: Ri <—> Rj) 

Type 2: Multiply a row by a nonzero constant. (Notation: cRi —» II,) 
Type 3: Add a multiple of one row to another row. (Notation: cRi + Rj —> 

Rj) 

Gaussian elimination is the process of reducing a matrix to its RREF using 
these row operations. Each of these operations is the respective analogue of 
the equation operations described above, and each can be realized by left 
matrix multiplication. We have the following. 

Type 1 



row i 


row j 


1 
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Notation: Rj <H> Rj 

Type 2 


1 


E2 — 



row i 


column i 


1 


Notation: cRi 

Type 3 


1 



E3 — 



c 


column 

i 


row j 


1 


1 


Notation: cRi + Rj, the abbreviated form of cRi + Rj Rj 


Example 2.2.1. The operations 


"2 10" 


0 2 1' 


0 2 1' 

0 2 1 

R\ <—y R 2 

2 1 0 

4i? 3 

2 1 0 

-10 2 

-1 

-10 2 

-1 

-4 0 8 
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can also be realized as 

■ o i o i r 2 io] ro 2i" 

Ri i—> R 2 : 100 021 = 210 

.0 0 1 J —1 0 2 J [ -1 0 2 _ 

' 100 ] [0 21 ] [0 21 " 

4 R 3 : 010 210=210 

004j |_ — 1 0 2 J [-4 08 

The operations 


2 

1 

0 ' 

—3i?i + R 2 

—>■ 

"2 10 " 

0 

2 

1 

-6 -1 1 

L-1 0 

2 

2 i?i + R 3 

3 2 2 


can be realized by the left matrix multiplications 

" 1 0 0 ] [ 1 0 0 ] [ 2 10 ] [2 1 0 " 

0 1 0 -3 1 0 0 2 1 = -6 -1 1 

_ 2 0 1 J L 0 0 1 J |_ —1 0 2 J [3 2 2 

Note there are two matrix multiplications them, one for each Type 3 ele¬ 
mentary operation. 

Row-reduced echelon form. To each A £ M m)n {E ) there is a canonical 
form also in M m ^ n (E) which may be obtained by row operations. Called the 
RREF, it has the following properties. 

(a) Each nonzero row has a 1 as the first nonzero entry (:= leading one). 

(b) All column entries above and below a leading one are zero. 

(c) All zero rows are at the bottom. 

(d) The leading one of one row is to the left of leading ones of all lower 
rows. 

Example 2.2.2. 

"1 2 0 0 -1" 

0 0 10 3 

0 0 0 1 0 

0 0 0 0 0 


B = 


is in RREF. 
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Theorem 2.2.3. Let A G M mjn (F). Then the RREF is necessarily unique. 

We defer the proof of this result. Let A G M m>n (F). Recall that the 
row space of A is the subspace of R n (or C n ) spanned by the rows of A. In 
symbols the row space is 


6(ri(T),... ,r m (A)). 

Proposition 2.2.1. For A G M mtn (F) the rows of its RREF span the rows 
space of A. 

Proof. First, we know the nonzero rows of the RREF are linearly indepen¬ 
dent. And all row operations are linear combinations of the rows. Therefore 
the row space generated from the RREF is contained in the row space of 
A. If the containment is proper. That is there is a row of A that is lin¬ 
early independent from the row space of the RREF, this is a contradiction 
because every row of A can be obtained by the inverse row operations from 
the RREF. □ 

Proposition 2.2.2. If A G M m ^ n (F) and a row operation is applied to A, 
then linearly dependent columns of A remain linearly dependent and linearly 
independent columns of A remain linearly independent. 

Proposition 2.2.3. The number of linearly independent columns of A G 
Mm : n(F) is the same as the number of leading ones in the RREF of A. 

Proof. Let S = {R ... ik} be the columns of the RREF of A having a lead¬ 
ing one. These columns of the RREF are linearly independent Thus these 
columns were originally linearly independent. If another column is linearly 
independent, this column of the RREF is linearly dependent on the columns 
with a leading one. This is a contradiction to the above proposition. □ 

Proof of Theorem 2.2.3. By the way the RREF is constructed, left-to-right, 
and top-to-bottom, it should be apparent that if the right most row of the 
RREF is removed, there results the RREF of the m x (n — 1) matrix formed 
from A by deleting the n th column. Similarly, if the bottom row of the 
RREF is removed there results a new matrix in RREF form, though not 
simply related to the original matrix A. 

To prove that the RREF is unique, we proceed by a double induction, 
first on the number of columns. We take it as given that for anmxl matrix 
the RREF is unique. It is either the zero m x 1 matrix, which would be 
the case if A was zero or the matrix with a 1 in the first row and zeros in 
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the other rows. Assume therefore that the RREF is unique if the number 
of columns is less than n. Assume there are two RREF forms, B\ and 
B 2 for A. Now the RREF of A is therefore unique through the (n — l) st 
columns. The only difference between the RREF’s B\ and Bo must occur 
in the n th column. Now proceed by induction on the number of nonzero 
rows. Assume that 4^0. If A has just one row, the RREF of A is simply 
the scalar multiple of A that makes the first nonzero column entry a one. 
Thus it is unique. If A = 0, the RREF is also zero. Assume now that 
the RREF is unique for matrices with less than m rows. By the comments 
above that the only difference between the RREF’s B\ and Bo can occur at 
the (m, ?r)-entry. That is (Bi) m ,n 7 ^ ( B 2 )m,n■ They are therefore not leading 
ones. (Why?) There is a leading one in the m th row, however, because it 
is a non zero row. Because the row spaces of Bi and B 2 are identical, this 
results in a contradiction, and therefore the (m, ?r)-entries must be equal. 
Finally, B\ = B 2 . This completes the induction. (Alternatively, the two 
systems pertaining to the RREF’s must have the same solution set to the 
system Ax = 0. With (-Bi) m , n 7 ^ ( B 2 ) m ,m it is easy to see that the solution 
sets to B\x = 0 and B 2 X = 0 must differ.) □ 


[A\b] = 


i,n cMld. b G Rm 

(or C n ). Define 

Oil 

. . Clin 

bi 


<221 • 

• • Cl2n 

b 2 


a m i 

• • ci rnn 

brri_ 



[A| 6 ] is called the augmented matrix of A by b. [A\b\ 6 Afm.n+i (F). The 
augmented matrix is a useful notation for finding the solution of systems 
using row operations. 


Identical to other definitions for solutions of equations, the equivalence 
of two systems is defined via the idea of equality of the solution set. 

Definition 2.2.3. Two linear systems Ax = b and Bx = c are called equiv¬ 
alent if one can be converted to the other by elementary equation opera¬ 
tions. 


It is easy to see that this implies the following 

Theorem 2.2.4. Two linear systems Ax = b and Bx = c are equivalent if 
and only if both [A| 6 ] and [B|c] have the same row reduced echelon form. 

We leave the prove to the reader. (See Exercise 23.) Note that the solution 
set need not be a single vector; it can be null or infinite. 
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2.3 Rank 

Definition 2.3.1. The rank of any matrix A, denote by r(A), is the di¬ 
mension of its column space. 

Proposition 2.3.1. (i) The rank of A equals the number of nonzero rows 
of the RREF of A, i.e. the number of leadinq ones. 

(ii) r(A) = r{A T ). 

Proof, (i) Follows from previous results. 

(ii) The number of linearly independent rows equals the number of lin¬ 
early independent columns. The number of linearly independent rows is 
the number of linearly independent columns of A T —by definition. Hence 
r(A) = r(A T ). □ 

Proposition 2.3.2. Let A <E M m>n (C) and b £ C m . Then Ax = b has a 
solution if and only if r(A) = r([A|6]) ; where [A\b] is the augmented matrix. 

Remark 2.3.1. Solutions may exist and may not. However, even if a so¬ 
lution exists, it may not be unique. Indeed if it is not unique, there is an 
infinity of solutions. 

Definition 2.3.2. When Ax = b has a solution we say the system is con¬ 
sistent. 

Naturally, in practical applications we want our systems to be consistent. 
When they are not, this can be an indicator that something is wrong with 
the underlying physical model. In mathematics, we also want consistent 
systems; they are usually far more interesting and offer richer environments 
for study. 

In addition to the column and row spaces, another space of great impor¬ 
tance is the so-called null space, the set of vectors x € R n for which Ax = 0. 
In contrast, when solving the simple single variable linear equation ax = b 
with a^Owe know there is always a unique solution x = 6/a. In solving 
even the simplest higher dimensional systems, the picture is not as clear. 

Definition 2.3.3. Let A € M m , n (F). The null space of A is defined to be 

Null(A) = {x € R n | Ax = 0}. 

It is a simple consequence of the linearity of matrix multiplication that 
Null(A) is a linear subspace of i? n . That is to say, Null (A) is closed under 
vector addition and scalar multiplication. In fact, A(x + y) = Ax + Ay = 
0 + 0 = 0, if x, y £ Null(A). Also, A(ax) = a Ax = 0, if x £ Null(A). We 
state this formally as 
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Theorem 2.3.1. Let A G M mjn (F). Then Null(A) is a subspace of R n 
while the range of A is in R rn . 


Having such solutions gives valuable information about the solution set 
of the linear system Ax = b. For, if we have found a solution, x, and have 
any vector z G Null(T), then x + z is a solution of the same linear system. 
Indeed, what is easy to see is that if u and v are both solutions to Ax = b, 
then A(u — v) = An — Av = 0, or what is the same x — y G Null(H). This 
means that to find all solutions to Ax = b, we need only find a single solution 
and the null space. We summarize this as the following theorem. 


Theorem 2.3.2. Let A G M m ^ n {F) with null space Null(A). Let x be any 
nonzero solution to Ax = b. Then the set x + Null(A) is the entire solution 
set to Ax = b. 


Example 2.3.1. Find the null space of A = 
Solution. Solve Ax = 0. The RREF for A is f 


3 

-9 ' 

. Solving xi+3®2 = 0, 


take X 2 = t, a “free” parameter and solve for x\ to get x\ = —3 1. Thus 
every solution to Ax = 0 can be written in the form 


t G R 


Expressed this way we see that Null(H) = 
of i ?2 of dimension 1. 

Theorem 2.3.3 (Fundamental theorem on rank). A G M mn (E). The 
following are equivalent 

(a) r{A) = k. 

(b) There exist exactly k linearly independent columns of A. 

(c) There exist exactly k linearly independent rows of A. 

(d) The dimension of the column space of A is k (i.e. dim {Range A) = k). 

(e) There exists a set S of exactly k vectors in R m for which Ax = b has 
a solution for each b G &(S). 

(f) The null space of A has dimension n — k. 


-3 

1 


t G R > , a subspace 
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Proof. The equivalence of (a), (b), (c) and (d) follow from previous con¬ 
siderations. To establish (e), let S = {c £ l , q 2 , ... , 04 } denote the linearly 
independent column vectors of A. Let T = {e^ 1 , e^ 2 ,... ,e£ k } C R n be the 
standard vectors. Then Ae^. = eg.. If b 6 6(5), then b = oqc^ + + 

• • • + a k C£ k . A solution to Ax = b is given by x = Qqe^ + <2264 + • • • + a k e£ k . 
Conversely, if (e) holds, then the set S must be linearly independent for 
otherwise S could be reduced to k — 1 or fewer vectors. Similarly if A has 
k + 1 linearly independent columns then set S can be expanded. Therefore, 
the column space of A must have exactly k vectors. 

To prove (f) we assume that S = {rq,... , v k } is a basis for the column 
space of A. Let T = {wq, ... , w k } C R n for which Awi =Vi, i = 1 ,... , k. 
By our extension theorem, we select n — k vectors w k +i ,... , w n such that 
U = {uq,... ,w k ,w k+ i,... ,w n } is a basis of R n . We must have that 
Awk+i £ ©(S'). Hence there are scalars b\,... , b}. such that 


Aw k+ 1 = A(biwi -)-f b k w k ) 

and thus w' k+l = "uq + i — (b±wi + • • • + b k w k ) is in the null space of A. 
Repeat this process for each w k+ j, j = 1,... ,n — k. We generate a total 
of n — k vectors {w;(, +1 ,... ,w' n } in this manner. This set must be linearly 
independent. (Why?) Therefore, the dimension of the null space must be 
at least 7i — k. Now we consider a new basis which consists of the original 
vectors and the n — k vectors {w' k+1 , w' k+2 ,... ,w' n } for which Aw = 0. We 
assert that the dimension of the null space is exactly n — k. For if z £ R n is 
a vector for which Az = 0, then z can be uniquely written as a component 
z\ from ©(T) and a component Z 2 from 6({u4 +1 ,... ,w' n }). But Az\ 0 
and Az 2 = 0. Therefore Az = 0 is impossible unless the component z\ = 0. 

Conversely, if (f) holds we take a basis for the null space T = {u\,U 2 , ... ,u n 
and extend the basis 


T' = TU{ti n _ w , •••, u n } 
to R n . Next argue similarly to above that 

fc+ 1 ) Au n - k -£ 2 , ■ ■ ■ ■ Au n 

must be linearly independent, for otherwise there is yet another linearly 
independent vector that can be added to its basis, a contradiction. Therefore 
the column space must have dimension at least, and hence equal to k. □ 

The following corollary assembles many consequences of this theorem. 
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Corollary 2.3.1. (1) r(A) < min(m,n). 

(2) r(AB ) < mm(r(A),r(B)). 

(3) r(A + B) < r{A) + r(B). 

(4) r(A) = r(A T ) = r(A*) = r(A). 

(5) If A £ M m (F) and B £ M m ^ n (F), and if A is invertible, then 

r(AB) = r(B). 

Similarly, if C £ M n (F) is invertible and B £ M rn n (F) 

r(BC) = r(B). 

(6) r{A) = r(A T A) = r{A*A). 

(7) Let A £ M mn (F), with r(A) = k. Then A = XBY where X £ M m ^, 
Y £ Mk t n and B £ Mk is invertible. 

(8) In particular, every rank 1 matrix has the form A = xy T , where x £ 
R rn and y £ R n . Here 


Xiyi Xiy 2 .. 

• • xiy n 

%m,y 1 %ml/2 • 1 

• • *£m2/n 


Proof. (1) The rank of any matrix is the number of linearly independent 
rows, which is the same as the number of linearly independent columns. 
The maximum this value can be is therefore the maximum of the 
minimum of the dimensions of the matrix, or r ( A ) < min (m, n). 


(2) The product AB can be viewed in two ways. The first is as a set 
of linear combinations of the rows of B, and the other is as a set of 
linear combinations of the columns of A. In either case the number 
of linear independent rows (or columns as the case may be) In other 
words, the rank of the product AB cannot be greater than the number 
of linearly independent columns of A nor greater than the number of 
linearly independent rows of B. Another way to express this is as 
r (AB) < min(r (A), r (. B )) 
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(3) Now let S = {vi,.. ■ v r (A)} and T = {tei,... ,w r ^)} be basis of the 
column spaces of A and B respectively. Then, the dimension of the 
union SUT = {vi,... v r rjo, W \,..., tc r (s)} cannot exceed r (A)+r ( B). 
Also, every vector in the column space of A + B is clearly in the span 
of SUT. The result follows. 


(4) The rank of A is the number of linearly independent rows (and columns) 
of A, which in turn is the number of linearly independent columns of 
A t , which in turn is the rank of A T . That is, r ( A ) = r (A T ) . Similar 
proofs hold for A* and A. 

(5) Now suppose that A G M m (F ) is invertible and B G M m ,n. As 
we have emphasized many times the rows of the product AB can be 
viewed as a set of linear combinations of the rows of B. Since A has 
rank m any set of linearly independent rows of B remains linearly 
independent. To see why, let r* (AB) denote the i th row of the product 
AB. Then it is easy to see that 

m 

n (AB) = ^2 a ij r j ( B ) 

3 =1 

Suppose we can determine constants cp ..,, c rn not all zero so that 

m mm 

0 = £ CjT-i (AB) = ^2 c i ^2 a b'0‘ ( B ) 

j= 1 2=1 3=1 

m m 

= r i ^ Ciai i 

j= 1 2=1 

This linear combination of the rows of B has coefficient given by A T c, 
where c = [cp ... ,Ck] T ■ Because the rank of A (and A T ) is m, we 
can solve this system for any vector d G R m . Suppose that the row 
vectors r-,-, (B ), l = 1 ,...,r(B), are linearly independent. Arrange 
that the components of d to be zero for indices not included in the set 
ji , l = 1 ,...,r(B) and not all zero otherwise. Then the conclusion 

0 = fj (B) YliL 1 c i a tj = r 3 i (B) dj t is impossible. Indeed, 

the same basis of the row space of B will be a basis of the row space 
of AB. This proves the result. 

(6) We postpone the proof of this result until we discuss orthogonality. 
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(7) Place A in RREF, say Arref- Since r (A) = k we know the top k 
rows of Arref are linearly independent and the remaining rows are 
zero. Define Y to be the k x n matrix consisting of these top k rows. 
Define B = R. Now the rows of A are linear combinations of these 
rows. So, define the m x k matrix X to have rows as follows: The 
first row of consists of the coefficients so that J2 x ij r j(X) = n (A). 
In general, the i th row of X is selected so that 

^XijrjiY) = n (A) 

(8) This is an application of (7) noting in this special case that X is an 
m x 1 matrix that can be interpreted as a vector x € R m . Similarly, 
Y is an 1 x n matrix that can be interpreted as a vector y e R n . 
Thus, with / = [1], we have 

A = xy T 


□ 


Example 2.3.2. Here is the decomposition of the form given in Le mm a 
2.3.1 (7). The 3x4 matrix A has rank 2. 


1 

2 

-1 ' 


1 

t-H 

0 

0 

2 


0 

2 

-1 

-2 

CO 


-1 

CO 

2 

4 

0 


2 

o J 


The matrix Y is the RREF of A. 


[1 

1 

o 


' 1 

2 

1- 

o 

i- 

o 

1 


o 

0 

Ij 


= XBY 


Example 2.3.3. Let x = [xi, x 2 , ■ ■ ■, x m ] T <E R m and y = [yi,y 2 , ■ ■ ■ y n ] T £ 
R n . Then the rank one m X n matrix xy T has the form 



xm 

x\y 2 ■■ 

• • xiy n 

T 

xy = 

x 2 yi 

x 2 y 2 

x 2 y n 


%m,yi 

x m y 2 • 1 

Xmyn 


In particular, with x = [1, 3, 5 ] T , and y = [—2, 7] T , the rank one 3x2 matrix 
xy T is given by 



■ i ■ 


—2 

7 

T 

xy = 

3 

[-2, 7] = 

—6 

21 


5 


-10 

35 
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Invertible Matrices 

A subclass matrices A £ M n (F ) that have only the zero kernel is very 
important in applications and theoretical developments. 

Definition 2.3.4. A £ M n is called nonsingular if Ax = 0 implies that 
x = 0. 

In many texts such matrices are introduced though an equivalent alter¬ 
nate definition involving rank. 

Definition 2.3.5. A £ M n is nonsingular if r(A ) = n. 

We also say that nonsingular matrices have full rank. That nonsingular 
matrices are invertible and conversely together with many other equivalences 
is the content of the next theorem. 

Theorem 2.3.4. [Fundamental theorem on inverses] Let A £ M n (F). Then 
the following statements are equivalent. 

(a) A is nonsingular. 

(b) A is invertible. 

(c) r(A) = n. 

(d) The rows and columns of A are linearly independent. 

(e) dim(Range(A)) = n. 

(f) dim(Null(A)) = 0. 

(g) Ax = b is consistent for all b £ R n (or C n ). 

(h) Ax = b has a unique solution for every x £ R n (or C n ). 

(i) Ax = 0 has only the zero solution. 

(j) * 0 is not an eigenvalue of A. 

(k) * detA^O. 

* The statements about eigenvalues and the determinant (det A) of a ma¬ 
trix will be clarified later after they have been properly defined. They are 
included now for completeness. 
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Definition 2.3.6. Two linear systems Ax = b and Bx = c are called equiv¬ 
alent if one can be converted to the other by elementary equation opera¬ 
tions. Equivalently, the systems are equivalent if [A \ b] can be converted to 
\B | c] by elementary row operations. 

Alternatively, the systems are equivalent if they have the same solution 
set which means of course that both can be reduced to the same RREF. 

Theorem 2.3.5. If A € M n (F) and B € M n (F ) with AB = I, then B is 
unique. 

Proof. If AB = I then for every e\ ... e n there is a solution to the system 
Abi = e* for all 1 = 1, 2, ... ,n. Thus the set {&i}f =1 is linearly independent 
(because the set {e*} is) and moreover a basis. Similarly if AC = I then 
A(C — B) = 0, and there are c* € R n (or C n ), i = 1, ..., n such that 
Act = Cj. Suppose for example that c\ — b\ 0. Since the {6j}™ =1 is a basis 
it follows that c\ — b\ = Eajbj, where not all ctj are zero. Therefore, 

A{c\ — b\) = EctjAbj = Eajej 0. 

and this is a contradiction. □ 

Theorem 2.3.6. Let A € M n (F). If B is a right inverse, AB = I, then B 
is a left inverse. 

Proof. Define C = BA — I + B, and assume C B or what is the same 
thing that B is not a left inverse. Then 

AC = ABA - A + AB = ( AB)(A ) - A + AB 
= A-A + AB = I 

This implies that C is another right inverse of A, contradicting Theorem 
2.3.5. □ 

2.4 Orthogonality 

Let V be a vector space over C. We define an inner product (•, •) on V x V 
to be a function from V to C that satisfies the following properties: 

1. ( av,w ) = a(v,w) and ( v,aw ) = a(v,w) (a is the complex conjugate 
of a ) 


2. (v,w) = (w,v ) 
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3. (u + v, w) = (u, w) + ( v , w) (linearity) 

4. ( u , v + w) = (u, v) + (u, ru) 

5. ( v , u) > 0 with (u, u) = 0 if and only if v = 0. 

For inner products over real vector spaces, we neglect the complex con¬ 
jugate operation. In addition, we want our inner products to define a norm 
as follows: 

6. For any v G V, ||v|| = (v,v) 


We assume thoughout the text that all vector spaces with inner products 
have norms defined exactly in this way. With the norm and vector v can 
be normalized by dilating it to have length 1, say v n = The simplest 

type of inner product on C n is given by 

n 

(v, w) = Xiyi 
i= 1 

We call this the standard inner product. 

Using any inner product, we can define an angle between vectors. 


Definition 2.4.1. The angle 9 xy between vectors x and y in R n is defined 
by 


COS 0 X y — 


(x,y) 

MUM 


(% •) y } 

((x,x)) 1 /2((y ) y))l/2' 


This comes from the well known result in R 2 


x ■ y = ||x|||MI cos# 

which can be proved using the law of cosines. With angle comes the notion 
of orthogonality. 

Definition 2.4.2. Two vectors u and v are said to be orthogonal if the 
angle between them if | or what is the same thing (u,v) = 0. In this case 
we commonly write il y. We extend this notation to sets U writing xLU 
to mean that xLu for every u € U. Similarly two sets U and V are called 
orthogonal if uLv for every u € U and v £ V. 
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Remark 2.4.1. It is important to note that the notion of orthogonality 
depends completely on the inner product. For example, the weighted inner 

n 

product defined by (v, w) = ^ w i x iVi where the Wi> 0 gives very different 

2—1 

orthogonal vectors from the standard inner product. 

Example 2.4.1. In R n or C n the standard unit vectors are orthogonal with 
respect to the standard inner product. 

Example 2.4.2. In the R3 the vectors u = (1,2,—1) and v = (1,1,3) are 
orthogonal because 

(x,y) = 1 (1) + 2 (2) - 1 (3) = 0 

Note that in R3 the complex conjugate is not written. The set of vectors 
(xi,X 2 ,xs) 6 R 3 orthogonal to u = (1,2,—1) satisfies the equation x\ + 
2x 2 — X3 = 0 is recognizable as the plane with normal vector u. 

Definition 2.4.3. We define the projection P u v of one vector v in the di¬ 
rection of an other vector u to be 


||u|| 

As you can see, we have merely written an expression for the more in¬ 
tuitive version of the projection in question given by ||u|| cos 9 UV . In the 
figure below, we show the fundamental diagram for the projection of one 
vector in the direction of another. 


v 



If the vectors u and v are orthogonal, it is easy to see that P u v = 0 . 
(Why?) 

Example 2.4.3. Find the projection of the vector v = (1, 2, 1) on the vector 
u = (-2,1,3) 
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Solution. We have 

p.v - M u= «hMA 
INI 2 IK-2, 1 

= 1( - 2) + 2 1 < 1> + 1(3) ( -2, 1 . 3) 

= n (_2,1 ’ 3) 

We are now ready to find orthogonal sets of vectors and orthogonal bases. 
First we make an important definition. 

Definition 2.4.4. Let V be a vector space with an inner product. A set 
of vectors S = {x \,... , x n } in V is said to be orthogonal if (xi, Xj) = 0 for 
i ^ j. It is called orthonormal if also {xi,xf) = 1. If, in addition, S' is a 
basis it is called an orthogonal basis or orthonomal basis. 

Note: Sometimes the conditions for orthonormality are written as 

Xi , Xj ) — Sjj 

where Sij is the “Dirac” delta: 8 ij = 0, i ^ j, Su = 1. 

Theorem 2.4.1. Suppose XJ is a subspace of the (inner product) vector 
space V and that U has the basis S = {x\ ... X]f\, then U has an orthogonal 
basis. 

Proof. Define y\ = Thus y\ is the “normalized” x\. Now define the 

new orthonormal basis recursively by 


-2,1,3)} 


> 3 )||" 


(-2,1,3) 


Vj +1 


Vj +1 = 


Xj+i - ^2(yi,x j+1 )yi 


2—1 


yj +1 

\W j+1 \\ 


for j = 1, 2,..., k — 1. Then 

(1) ]Jj+i is orthogonal to y±, 

(2) Uj +1 ~f~ 0. 


Vj 


In the language above we have (yi,yj) = <%• 


□ 
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Basically, what the proof accomplishes is to take the differences of the 
vector from the projections to the others. Referring to the figure above we 
compute v — P u v as noted in the figure below. The process of orthogonal- 
ization described above is called the Gram—Schmidt process. 


v 



Representation of vectors 

One of the great advantages of orthonormal bases is that they make the 
representation of vectors particularly easy. It is as simple as computing an 
inner product. Let V be a vector space with inner product (•, •} and with 
subspace U having basis S = {ui, u 2 , ■ ■ •, u k }. Then for every u & U we 
know there are constants ai, a 2 ,..., a k such that 


x = aiui + a 2 u 2 H-(- a k u k . 


Taking the inner product of both sides with Uj and applying the orthogo¬ 
nality relations 


(x, Uj) = (aiui + a 2 u 2 H-b a k Uk.,Uj) 

k 

— ^ ^ C i (U-L ■ Uj } — Uj 

3 = 1 

Thus aj = (x, Uj), j = 1,2, ..., k, and 

k 

X = (u.,Uj) Uj 

3 = 1 


Example 2.4.4 

{ui,u 2 }, where u\ = 
tion of x = [3, 2 ] T is 


One basis of R 2 is given by the orthonormal vectors S = 

iT 

The representa- 


1 1 
-v/2’ %/2_ 

T 

and u 2 = 

l_i_ 

V2’ %/2_ 

given by 




2 

X = (u.,Uj) Uj 

3 = 1 



' 1 1 ‘ 

T i 

' 1 1 ' 

. 71 ’ Vi 

^ 2 V 

. 71 ’ “ 71 . 
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Orthogonal subspaces 

Definition 2.4.5. For any set of vectors S we define 

S ± = {u € V | vLS} 

That is, S 1 - is the set of vectors orthogonal to S. Often, S 1 - is called the 

orthogonal complement or orthocomplement of S. 

For example the orthocomplement of any vector v = \vi,V 2 , V 3 ] T GE R 3 is the 
(unique) plane passing through the origin that is orthogonal to v. It is easy 
to see that the equation of the plane is x\V\ + X 2 V 2 + X 3 V 3 = 0. 

For any set of vectors S the orthocomplement S ^ has the remarkable 
property of being a subspace of V, and therefore it is must have an orthog¬ 
onal basis. 

Proposition 2.4.1. Suppose that V is a vector space with an inner product, 
and S C V. Then S 1 - is a sub space of V. 

Proof. If ;yi, ... , y m e S 1 - then Ea*yj € S ± for every set of coefficients 
ai, ... , a m in R (or C). □ 

Corollary 2.4.1. Suppose that V is a vector space with an inner product, 
and S C V. 

(i) If S is a basis ofV, S 1 = {0}. 

(ii) IfU = &(S), then U x = S ± . 

The proofs of these facts are elementary consequences of the proposition. 
An important decomposition result is based on orthogonality of subspaces. 
For example, suppose that V is a finite dimensional inner product space and 
that U is a subspace of V. Let U 1 be the orthocomplement of U, and let 
S = {ui,U 2 , ■ ■ ■ ,Uk} be an orthonormal basis of U. Let x G V. Define 
X\ = Ylj=i { x i u j) u ji and X 2 = x — X\. Then it follows that x\ € U and 
X2 6 . Moreover, x = x\ + X2- We summarize this in the following. 

Proposition 2.4.2. Let V is a vector space with inner product (•,•) and 
with subspace U. Then every vector x € V can be written as a sum of two 
orthogonal vectors x = x\ + X 2 , where x\ € U and X 2 € U L . 
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Geometrically what this results asserts is that for a given subspace of 
an inner product space, every vector has an orthogonal decomposition as 
two unique sum of a vector from the subspace and its orthocomplement. 
We write the vector components as the respective projections of the given 
vector to the orthogonal subspaces 


X\ = Pjj x 
X 2 = Pjj±X 

Such decompositions are important in the analysis of vector spaces and 
matrices. In the case of vector spaces, of course, the representation of 
vectors is of great value. In the case of matrices, this type of decomposition 
serves to allow reductions of the matrices while preserving the information 
they carry. 

2.4.1 An important equality for matrix multiplication and 
the inner product 

Let A € M mn (C). Then we know that both A* A and AA* (Alternatively, 
A T A and AA T exist) exist, and we can surely inquire about the rank of these 
matrices. The main result of this section is on the rank of A T A, namely 
that r (A) = r(A*A ) = r(AA*). The proof is quite simple but requires an 
important equality. Let A e M mn (C) and v € C n and w € C m . Then 


(Av, w) 



m n 

Y a B V 3 
2=1 i i 


n m 

Y v i 

j = 1 *=1 

n m 

j =1 *=1 

n 

Vj ( A *w)j 

3 =1 

(v, A*w) 
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As a consequence we have ( A*Av,w } = ( Av,Aw ) if both v, w € C n . This 
important equality allows the adjoint or transpose matrices to be used on 
either side of the inner product, as needed. Indeed we shall use this below. 

Proposition 2.4.3. Let A £ M mn ( C) have rank r ( A ). Then 

r (A) = r(A*A) = r(AA*) 

Proof. Assume that r (A) = k. Then there are k standard vectors ej 1 ,..., ej k 
such that for each l = 1 , 2 ,...k, the vectors Ae^ is one of the linearly 
independent columns of A. Moreover, it also follows that for every set of 
constants a\,...,ak the vector 7 ^ 0. Now A*A a l e lj) 7 ^ 0 

follows because 

’ A {Y. aie h)) 

= \ A (j2 a i e h)\\ 

This in turn establishes that A* A cannot be zero on a linear space of di¬ 
mension k except for the zero element of course, and since the rank of A* A 
cannot be larger than k the result is proved. □ 

Remark 2.4.2. This establishes ( 6 ) of the Corollary 2.3.1 above. Also, it 
is easy to see that the result is also true for real matrices. 

2.4.2 The Legendre Polynomials 

When a vector space has an inner product, it is possible to construct an 
orthogonal basis from any given basis. We do this now for the polynomial 
space P n (—1,1) and a particular basis. 

Consider the space the polynomials of degree n defined on the interval 
[—1,1] over the reals. Recall that this is a vector space and has as a basis 
the monomials {l, x, x 2 , ..., cc n } . We can define an assortment of inner 
products on this space, but the most common inner product is given by 

(P, q) = J P{x)q (x) dx 

Verifying the inner product properties is fairly straight forward and we leave 
it as an exercise. This inner product also defines a norm 

\\p \\ 2 = f Jp(x )\ 2 dx 
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This norm satisfies the triangle inequality requires an integral version of the 
Cauchy-Schwartz inequality. 

Now that we have an inner product and norm, we could proceed to find 
an othogonal basis of P n (—1,1) by applying the Gram-Schmidt procedure 
to the basis {l, x, x 2 ,... , x n }. This procedure can be clumsy and tedious. 
It is easier to build an orthogonal basis from scratch. Following tradition 
we will use capital letters Pq,Pi,. .. to denote our orthogonal polynomials. 
Toward this end take Pq = 1. Note we are numbering from 0 onwards so 
that the polynomial degree will agree with the index. Now let P\ = ax + b. 
For orthogonality, we need 

J To (x) Pi (x) dx = J 1 • (ax + 6) dx = 2b = 0 


Thus 6 = 0 and a can be arbitrary. We take a = 1. This gives y\ = x. Now 
we assume the model for the next orthogonal function to be y 2 = ax 2 +bx+c. 
This time there are two orthogonality conditions to satisfy. 



P 0 ( X ) P 2 (x) dx 
Pi (x) P 2 (x) dx 



(ax 2 + bx + c) dx = -a + 2c = 0 
o 

2 

(ax 2 + bx + c) dx = -6 = 0 
o 


We conclude that 6 = 0. From the equation |a + 2c = 0, we can assign one 
of the variables and solve for the other one. Following tradition we take 
c = —\ and solve for a to get a = 

The next polynomial will be modeled as P 3 (x) = ax 3 + bx 2 + cx + d. 
Three orthogonality relations need to be satisfied. 



P 0 (x ) P 3 (x) dx 
Pi (x) P 3 (x) dx 
P 2 (x) P 3 (x) dx 



2 

1 • (ax 3 + bx 2 + cx + d) dx = -6 + 2 d = 0 

o 

,32 \ 22 
x ■ (ax + bx + cx + d) dx = -a + -c = 0 

5 3 

^ (3x — 1) (ax 3 + bx 2 + cx + d) dx 



d = 0 


It is easy to see that 6 


d = 0 (why?) and from |a + |c = 0, we select 
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c = — | and a = |. Our table of orthogonal polynomials so far is 


k 

Pk (x) 

0 

1 

1 

X 

2 

\ (3® - 1) 

3 

1 (5x 3 — 3x) 


Continue in this fashion, generating polynomials of increasing order each 
orthogonal to all of the lower order ones. 


Po(x) 
Pl(x) 
P‘2 (x) 
Pi,x) 

Pi{x) 

P 5 (x) 

Pdx) 

Pr(x) 

P*(x) 

P*(x) 

Pio(x) 


1 


X 


5/2 x 3 
35 4 

63 5 

— x 5 


3/2 x 

^ + 3/8 

35 , 15 

— x +—x 

105 2 5 


315 


x 4 + 


231 6 _ 

16 X 16 

429 7 693 = 

- x - x + 

16 16 

6435 o 3003 

-x 8 - x 6 + 

32 


16 X ’ 

315 , 

- x 

16 


16 

35 

16 


x 


128 
12155 q 

128 X 
46189 ,10 
256 X 


6435 

32 

109395 

256 


x 7 + 


x 


3465 a Ole) o 

~M X + 

9009 


315 

"32 
1155 

~Y2 


35 

128 


~W X ~~32~ X + ' 
45045 . 15015 

J _ X °-' 

128 128 ’ 


63 

256 


2.4.3 Orthogonal matrices 

Besides sets of vectors being orthogonal, there is also a definition of orthog¬ 
onal matrices. The two notions are closely linked. 

Definition 2.4.6. We say a matrix A e M n (C) is orthogonal if A*A = I. 
The same definition applies to matrices A € M n (R) with A* replaced by 
,4 T . 
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For example, the rotation matrices (Exercise ??) Bg = 


cos 6 — sin 0 

sin 0 cos 0 


are all orthogonal. 

A simple consequence of this definition is that the rows and the columns 
of A are orthonormal. We see for example that when A is orthogonal then 
(A*) 2 = (A -1 ) 2 = A~ 1 A~ 1 = (A 2 ) -1 . Such a definition applies, as well to 
higher powers. For instance, if A is orthogonal then A m is orthogonal for 
every positive integer m. 

One way to generate orthogonal matrices in C n (or R n ) is to begin with 
an orthonormal basis and arrange it into an n x n matrix either as its columns 
or rows. 


Theorem 2.4.2. (i) Let {xi}, i = 1, ..., n be an orthonormal basis of C n 
or (Rn ). Then the matrices 


u = 

" Xi • 
1 • 

• • X n ' 
•• 1 

and V = 

x\ — 

> • 


- 

- 


_ x n — 

> • 


formed by arranging the vectors Xi as its respective columns or rows are 
orthogonal. 

(ii) Conversely, U is an orthogonal matrix, the sets of its rows and 
columns are each orthonormal, and moreover each forms a basis of C n or 
(Rn)- 

The proofs are entirely trivial. We shall consider these types of results 
in more detail later in Chapter 4. In the meantime there are a few more 
interesting results that are direct consequences of the definition and facts 
about the transpose (adjoint). 

Theorem 2.4.3. Let A, B e M n ( C ) (or M n (R) ) be orthogonal matrices. 
Then 

(a) A is invertible and A -1 = A*. 

(b) For each integer k = 0, ±1, ±2,..., both A k and —A k are orthogonal. 

(c) AB is orthogonal. 

2.5 Determinants 

This section is about determinants that can be regarded as a measure of 
singularity of a matrix. More generally, in many applied situations that 
deal with complex objects, a single number is sought that will in some way 
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classify an aspect of those objects. The determinant is such a measure for 
singularity of the matrix. The determinant is difficult to calculate and of 
not much practical use. However, it has considerable theoretical value and 
certainly has a place of historical interest. 

Definition 2.5.1. Let A G M n (F). Define the determinant of A to be 
the value in F 


det A = ( II a Mi)J ' s S n a 

where a is a permutation of the integers { 1 , 2 ,... , n} and 

( 1 ) ^2 denotes the sum over all permutations 

cr 

( 2 ) sgn a = sign of cr = ±1 

A transposition is the exchange of two elements of an ordered list with 
all others staying the same. With respect to permutations, a transposition 
of one permutation is another permutation formed by the exchange of two 
values. For example a transposition of {1,4, 3, 2} is {1,3,4,2}. The sign of 
a given permutation a is 

(a) + 1 , if the number of transpositions required to bring a to { 1 , 2 ,... , n} 
is even. 

(b) — 1 , if the number of transpositions required to bring a to { 1 , 2 ,... , n} 
is odd. 

Alternatively, and what is the same thing, we may count the number m of 
transpositions required to bring a to { 1 , 2 ,... , n} and to compute the sign 
is (-l) m . 

Example 2.5.1. 

(7i ={2,1,3} ^4 {1,2,3} odd 

(72 = {2,3,1} ^4 {2,1,3} -^4 {1,2,3} even 
sgn (7 1 = -1 sgn cr 2 = +1 


Proposition 2.5.1. Let A G M n . 
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(i) If two rows of A are interchanged to obtain B, then 

det-B = — det A 

(ii) Given A e M n (F). If any row is multiplied by a scalar c, the resulting 
matrix B has determinant 


det-B = cdet A. 

(Hi) If any two rows of A € M n (F) are equal, 

det A = 0. 

Proof. (i) Suppose rows i\ and i 2 are interchanged. Now for the given 
permutations a apply the transposition i\ ■(->■ i 2 to get o\. Then 

n n 

= (i) 

i —1 i =1 

because 

a il<j(il) — ^2Cl(*2) 

as 

bi 2 j = a h j and oi(* 2 ) = ^(k) 

and similarly Oi 2<r (i 2 ) = b iiai ^ 1 ). All other terms are equal. In the 
computation of the full determinant with signs of the permutations, 
we see that the change is caused only by the fact sgn(ai) = —sgn(cr). 
Thus, 


det-B = — det A. 


(ii) Is trivial. 

(iii) If two rows are equal then by part (i) 

det A = — det A 


and this implies det A = 0. 

□ 
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Corollary 2.5.1. Let A € M n . If A has two rows equal up to a multiplica¬ 
tive constant, it has has determinant zero. 

What happens to the determinant when two matrices are added. The 
result is too complicated to write down is not very important. However, 
when a single vector is added to a row or a column of a matrix, then the 
result can be simply stated. 

Proposition 2.5.2. Suppose A 6 M n (F). Suppose B is obtained from A 
by adding a vector v to a given row (resp. column) and C is obtained from 
A by replacing the given row (resp. column) by the vector v. Then 

det B = det A + det C. 

Proof. Assume the j th row is altered. Using the definition of the determi¬ 
nant, 


det A = Y sgn (a) b ia ^ = Y sgn (a) 






Y S9n II ( a + v ). 




1*7 U 


Y sgn ^ U a Mi) I a MA + Y sgn ^ ( n a Mi) I V MJ) 

cr \i^j ) a \i^j 

det A + det C 


For column replacement the proof is similar, particularly using the alternate 
representation of the determinant given in Exercise 15. □ 

Corollary 2.5.2. Suppose A € M n (F) and B is obtained by multiplying a 
given row (resp. column) of A by a scalar and adding it to another row 
(resp. column), then 


det B = det A. 

Proof. First note that in applying Proposition 2.5.2 C has two rows equal 
up to a multiplicative constant. Thus det C = 0. □ 

Computing determinants is usually difficult and many techniques have 
been devised to compute them out. As is evident from counting, computing 
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the determinant of an n x n matrix using the definition above would require 
the expression of all n! permutations of the integers {1, 2,, n} and the 
determination of their signs together with all the concommitant products 
and summation. This method is prohibitively costly. Using elementary 
row operations and Gaussian elimination, the evaluation of the determinant 
becomes more manageable. First we need the result below. 

Theorem 2.5.1. For the elementary matrices the following results hold. 

(a) for Type 1 (row interchange) E\ 

det Ei =s — 1 

(b) for Type 2 (multiply a row by a constant c) E 2 

det E 2 = c 

(c) for Type 3 (add a multiple of one row to another row) E 3 

det E% = 1. 

Note that (c) is a consequence of Corollary 2.5.2. Proof of parts (a) and (b) 
are left as exercises. Thus for any matrix A € M n {F) we have 

det (Ed A) = — det A = det E\ det A 
det(£ ! 2-4) = c det A = det E 2 det A 
det(£ 3 A) = det A = det E% det A. 

Suppose Fi... Fk is a sequence of row operations to reduce A to its RREF. 
Then 


Now we see that 


FkFk-i ■ ■ ■ F\A — B. 


det B = det(FfcF) c _ 1 ... F\A) 

= det(Ffc)det(Ffe_i ...FiA) 


= det(.Ffc) det(Ffe_i)... det(Fi) det A. 

For B in RREF and B G M n (F), we have that B is upper triangular. The 
next result establishes Theorem 2.3.4(k) about the determinant of singular 
and non singular matrices. Moreover, the determinant of triangular matrices 
is computed simply as the product of its diagonal elements. 
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Proposition 2.5.3. Let A £ M n . Then 

(i) If r(A) < n, then det A = 0. 

(ii) If A is triangular then 

det A = an 

(Hi) Ifr(A) = n, then det A 0. 

Proof, (i) If r{A) < n, then its RREF has a row of zeros, and det A = 0 by 
Theorem 2.5.1. (ii) If A is triangular the only product without possible zero 
entries is an. Hence det A — da- (hi) If If r(A) = n, then its RREF has 
no nonzero rows. Since it is square and has a leading one in each column, 
it follows that the RREF is the identity matrix. Therefore det A 0. □ 

Now let A, B 6 M n . If A is singular the RREF must have a zero row. 
It follows that det A = 0. If A is singular it follows that AB is singular. 
Therefore 


0 = det AB = det A det B. 

The same reasoning applies if B is singular. If A and B are not singular 
both A and B can be row reduced to the identity. Let F\... F kl be the row 
operations that reduce A to I, and G i... G kg be the row operations that 
reduce B to /. Then 


det A = [det(Ei)... det(Efc A )] 1 
det B = [det(Gi)... det(Gfc fl )] -1 . 

Also 

I = (G kB ...Gi)(F kB ...F 1 )AB 


and we have 


det I = (det A ) 1 (det B) 1 det AB. 


This proves the 

Theorem 2.5.2. If A, B € M n (F), detAB = det A det B. 
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2.5.1 Minors and Determinants 

The method of row reduction is one of the simplest methods to compute 
the determinant of a matrix. Indeed, it is not necessary to use Type 2 el¬ 
ementary transformation. This results in the computing the determinant 
as the product of the diagonal elements of the resulting triangular matrix 
possibly multiplied by a minus sign. An alternate approach to computing 
determinants using minors is both interesting and useful. However, unless 
the matrix has some special form, it does not provide a computational al¬ 
ternative to row reduction. 

Definition 2.5.2. Let A e M n (C ). For any row i and column j define the 
(ij)-minor of A by 


The notation 


M t j = det A 


jth 


row removed 


j * column removed 


A 


row removed 


column removed 


denotes the (n — 1) x (n — 1) matrix formed from A by removing the i th row 
and j th column. With minors an alternative formulation of the determinant 
can be given. This method, while not of great value computationally, has 
some theoretical importance. For example, the inverse of a matrix can be 
expressed using minors. We begin by consideration the determinant. 


Theorem 2.5.3. Let A € M n ( C ). (i) Fix any row, say row k. The de¬ 
terminant of A is given by 


n 

det A = ^ a kj (-l) k+j M kj 
j =i 

(ii) Fix any column, say column m. The determinant of A is given by 

n 

det A = Y J Hm (—1 ) m+j M jm 

3 = 1 


Proof, (i) Suppose that k = 1. Consider the quantity 


an Mn 
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We observe that this is equivalent to all the products of the form 

sgn (a) an ■ a 2a { 2 ) ' '' ^Wr(n) 

where only permutations that fix the integer (i.e. position) 1 are taken. 
Thus a (1) = 1. Since this position is fixed the signs taken in the determi¬ 
nant Mu for permutations of n — 1 integers are respectively the same as the 
signs for the new permutation of n integers. 

Now consider all permutations that fix the integer 2 in the sense that 
a (1) = 2. The quantity a \2 (—1) 1+2 M \ 2 consists of all the products of the 
form 

Sgn (cr) 0.12 ' ®2 <t(1)®3i 7(3).®nc(n) 

We need here the extra sign change because if the part of the permutation 
a of the integers { 1 , 3 , 4 , ... ,n} is of one sign, which is the sign used in 
the computation of det M- -, then the permutation of a of the integers 
{1, 2, 3 , 4 ,. .., n} is of the other sign, and that sign is sgn (cr). 

When we proceed to the fc th component, we consider permutations that 
fix the integer k. That is, cr (1) == k. In this case the quantity (—l) 1+fc M ik 
consists of all products of the form 

Sgn (cr) fllfcQ2er(l) " ^k—lcr(k—l)^k-\-la(k+l) .® rw(n) 

Continuing in this way we exhaust all possible products • a 2a ( 2 ) . 

a ncr(n) over all possible permutations of the integers {1,2, This 

proves the assertion. The proof for expanding from any row is similar, with 
only a possible change of sign needed, which is a prescribed. 

(ii) The proof is similar. □ 


Example 2.5.2. Find the determinant of 


A = 


3 2—1 

0 1 3 

1 2 -1 


expanding across the first row and then expanding down the second column. 
Solution. Expanding across the first row gives 


det A 


an AIii ~ 0.12M12 + ai%Mi 2 


3 det 


— 2 det 


1 3 

2 -1 

3 ( — 7) — 2 (—3) — (—1) 


0 

1 


-14 


3 

-1 


— 1 det 


0 1 
1 2 







74 


CHAPTER 2. MATRICES AND LINEAR ALGEBRA 


Expanding across the second column gives 


det A 


—2 det 


0 3 

1 -1 


+ 1 det 


3 -1 
1 -1 


—2 (—3) + (—2) — 2 (9) = —14 


2 det 


3 -1 
0 3 


The inverse of the matrix can be formulated in terms of minors, which 
is formulated below. 


Definition 2.5.3. Let A e M n (C) (or M n (R)). Define the adjugate (or 
adjoint) matrix A by 

Aij = (- l) i+j Mji 


where Mji is the ji minor. 

The adjugate has traditionally been called the “adjoint”, but that terminol¬ 
ogy is somewhat ambiguous in light of the previous definition as complex 
conjugate transpose. Note that it is defined for all square matrices; when 
restricted to invertible matrices the inverse appears. 

Theorem 2.5.4. Let A € M n (C) (or M n (R)) be invertible. Then A -1 = 

-A—A 

det A A 

Proof. A quick examination of the ij-entry of the product AA yields the 
following sum 

n n 

a ijAjk = det 1) M kj 

There are two possibilities. (1) If i = k, then the summation above is the 
summation to form the determinant as described in Theorem 2.5.3. (2) If 

i k, the summation is the computation of the determinant of the matrix 
A with the A; th row replaced by the z th row. Thus the determinant of a 
matrix with two identical rows is represented above and this must be zero. 
We conclude that (aa) . ~ $ij > the usual Kronecker ‘delta,’ and the result 
is proved. □ 


Example 2.5.1. Find the adjugate and inverse of 


A 


2 4 
2 1 
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It is easy to see that 


A = 



-4 

2 


Also det A = —6. Therefore, the inverse 


A " 1 



-4 

2 


Remark 2.5.1. The notation for cofactors of a square matrix A is often 
used 

at, = (-iy u Mji 

Note the reversed order of the subscripts ij and then ji above. 


Cramer’s Rule 

We know now that the solution to the system Ax = b is given by x = A~ 1 b. 

A 

Moreover, the inverse A " 1 is given by A -1 = -where A is the adjugate 

det A 

matrix. The the component of the solution vector is therefore 


Xi 


det A 2 

3 = 1 
n 

-—-V (-i y+’Mjibj 
det A ^ v J 3 

3 = 1 

det A* 
det A 


where we define the matrix A, to be the modification to A by replacing its 
i th column by the vector b. In this way we obtain a very compact formula 
for the solution of a linear system. Called Cramer’s rule we state this 
conclusion as 


Theorem 2.5.1. (Cramer’s Rule.) Let A e M n (C) be invertible and b € 
C n . For each i = 1,, n, define the matrix Aj to be the modification of A 
by replacing its i th column by the vector b. Then the solution to the linear 


system Ax = b is given by components Xi 


det Aj 
det A ’ 


i = 1,, n. 
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Example 2.5.2. Given the matrix A 


2 4 
2 1 


and the vector b = 


. Solve the system Ax = b by Cramer’s rule. 


We have 


A\ = 


2 

-1 


4 

1 


and A 2 


2 2 
2 -1 


and det A\ = 6, det Ao = —6, det A = — 6. Therefore 


x\ = —1 and X 2 = 1 


A curious formula 

The useful formula using cofactors given below will have some consequence 
when we study positive definite operators in Chapter ??. 

Proposition 2.5.1. Consider the matrix 



0 

Xl 

x 2 ■ 

• • x n 


Xl 

ail 

012 • 

^1 n 

B = 

X2 

021 

022 • 

n 


X n 

®nl 

0?i2 

Q"nn 


Then 


det B = — dijXiXj 
where dij is the ij-cofactor of A. 

Proof. Expand by minors along the top row to get 

det B = Y^ (-1 ) j XjMy (B) 

Now expand the matrix of M±j ( B ) down the first column. This gives 

M 1 j{B) = Y J {-V i ~ 1 x i Mij{A) 
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Combining we obtain 
det B = 


£(-l VxjMuiB) 

E x i [E (- 1 )*” 1 x i M a ( A ) 

E E (- 1 )* +J_1 XjXiMij (A) 


-EE 


Q*ij %i 


□ 

The reader may note that in the last line of the equation above, we should 
have used a t j. However, the formulation given is correct, as well. (Why?) 


2.6 Partitioned Matrices 

It is convenient to study partitioned or “blocked” matrices, or more graph¬ 
ically said, matrices whose entries are themselves matrices. For example, 
with I 2 denoting the 2x2 identity matrix we can create the 4x4 matrix 
written in partitioned form and expanded form. 

a 0 6 0 

0/2 c /2 _ 0 a 0 6 

c /2 dL> c 0 <7 0 

0 c 0 d 

Partitioning matrices allows our attention to focus on certain structural 
properties. In many applications partititioned matrices appear in a natural 
way, with the particular blocks having some system context. Many similar 
subclasses and processes apply to partitioned matrices. In specific situations 
they can be added, multiplied, and inverted, just like regular matrices. It is 
even possible to perform “blocked” version of Gaussian elimination. In the 
few results here, we touch on some of these possibilities. 

Definition 2.6.1. For each 1 < i < m and 1 < j < n. let A t] be an m, x rij 
matrices where . Then the matrix 

An A\2 ■ ■ ■ Ai n 

A 21 A 2n ■ ■ ■ A 2n 

A = 

A m i A m 2 • • • A rnn 
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is a partitioned matrix of order (X m) i x (X/ n j) ■ 

The usual operations of addition and multiplication of partitioned ma¬ 
trices can be performed provided each of the operations makes sense. For 
addition of two partitioned matrices A and B it is necessary to have the 


same numbers of blocks of the 

respective same sizes. 

Then 



An A12 

A\ n 


' B n 

B\2 ' ' ' 

Bln 

A + B = 

A-2i A 2n 

■ • • A 2n 

+ 

B 2 i 

B2n ■ ■ ■ 

F>2n 


Ami A m 2 

• • • A 

ST-rnn 


Bml 

Bm2 

Bmn 


Au + Bu 

A\2 + B\2 

Al n 

+ Bi n 


— 

A'2 i + B 2 1 

A 2n + B 2n 

A2n 

+ B 2n 



Ami T B m 1 

A m 2 T B m 2 

■ ■ A 

mn 

T B m n 



For multiplication, the situation is a bit more complicated. For definiteness, 
suppose that B is a partitioned matrix with block sizes Si x tj, where 1 < 
i <p and 1 < j < q The usual operations to construct C = AB, 

n 

yy A-ijBjk 

3 = 1 

then make sense provided p = n and iij = Sj, 1 < j < n. 

A special category of partitioned matrices are the so-called quasi-triangular 
matrices, wherein Ajj = 0 if i > j for the “lower” triangular version. The 
special subclass of quasi-triangular matrices wherein Aij = 0 if i ^ j are 
called quasi-diagonal. In the case of the multiplication of partitioned 
matrices (C = AB) with the left multiplicand A a quasi-diagonal matrix, 
we have Cik = A^B Thus the multiplication is similar in form to the usual 
multiplication of matrices where the left multiplicand is a diagonal matrix. 

In the case of the multiplication of partitioned matrices ( C = AB) with the 
right multiplicand B a quasi-diagonal matrix, we have Cik = F° r 

quasi-triangular matrices with square diagonal blocks, there is an interesting 
result about the determinant. 

Theorem 2.6.1. Let A be a quasi-triangular matrix, where the diagonal 
blocks An are square. Then 

det A = det A vl 
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Proof. Apply row operations on each vertical block without row interchanges 
between blocks, without any Type 2 operations. The resulting matrix in 
each diagonal block position (i, i) is triangular. Be sure to multiply one 
of the diagonal entries by ±1, reflecting the number of row interchanges 
within a block. The resulting matrix can still be regarded as partitioned, 
though the diagonal blocks are now actually upper triangular. Now apply 
Proposition 2.5.3, noting that the product of each of the diagonal entries 
pertaining to the i th block is in fact det A tl . □ 


A simple consequence of this result, proved ala Gaussian elimination, is 
contained in the following corollary. 

Corollary 2.6.1. Consider the partitioned matrix 


An A12 

A21 A22 


with square diagonal blocks and with An invertible. Then the rank of A is 
the same as the rank of An if and only if A 22 = A 2 iA 11 1 Ai 2 . 

Proof. Multiplication of A by the elementary partitioned matrix 


yields 


E = 


I 0 
— A2lA 11 1 I 


EA 


I 

— A2lA 11 1 



\ An 

A12 


to 

A22 


An A12 

0 A22 — A2lA 11 1 Ai2 


Since E has full rank, it follows that rank(TJA) = rankA. Since EA is 
quasi-triangular, it follows that the rank of A is the same as the rank of An 
if and only if A 22 — A 2 iA^ 1 1 Ai 2 =0. □ 


2.7 Linear Transformations 

Definition 2.7.1. A mapping T from R n to R rn is called a linear trans¬ 
formation if 


T(x + y) = Tx + Ty 
T(ax) = aTx 


Vx, y G R n 
Va € F. 
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Note: We normally write Tx instead of T{x). 

Example 2.7.1. T: R n —» R rn . Let a £ R m and y £ R n ■ Then for each 
x £ R n , Tx = (x,y)a is a linear transformation. Let S = {vi...v n } be 
a basis of R n , and define the m x n matrix with columns given by the 
coordinates of Tv\, Tv 2 , ■ ■ ■ ,Tv n . Then this matrix 


A 


Tv i Tv 2 

I I 


Tv n 


is the matrix representation of T with respect to the basis S. Thus, if 
x = EaiVi, whence [x],s = (a± ... a n ), we have 


[Tx] s = A[x]s 


■k There is a duality between all linear transformations from R n to R m 
and the set M m , n (F). 

Note that M m ^ n (F) is itself a vector space over F. Hence C(F n ,F m ), 
the set of linear transformations from F n to F m is likewise. As such it has 
subspaces. 

Example 2.7.2. (1) Let x £ F n . Definite J = {T £ C \ Tx = 0}. Then 

J is a subspace of C(F n , F m ). 

(2) Let U = {T £ C{R n ,R n ) \ Tx > 0 if x > 0}, where {x > 0} means 
the positive orthant of R n . U is not a linear subspace of C(R n , R n ), 
though it is a convex set. 

(3) Define T: P n —> P n by Tp = T is a linear transformation. 

Example 2.7.3. Express the linear transformation D : P>, —> P$ given by 
Dp = -^p(x) as a matrix with respect to the basis. S = {1, x, x 2 , x 3 }. We 
have D1 = 0 = 0 + Ox + Ox 2 + Ox 3 . Also 

[Dl] s = [0,0,0, 0] T 


[Dx]s = [1,0, 0, 0] T 
[Dx 2 ] s = [ 0,2,0,0] t 
[Dx 3 ]s = [0, 0, 3, 0] T . 


similarly 
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Hence 


[D}s 


0 10 0 
0 0 2 0 
0 0 0 3 
0 0 0 0 


In this context the differentiation operator is rather simple. 

Example 2.7.4. Consider the linear transformation T defined by Tq = 

3 x-^q + x 2 q for q G P 2 . Find the matrix representation of T. 

Solution. First off we notice that this transformation has range in 
-P 4 . Let’s use the standard bases for this problem. We then determine 
the coordinates of T for vectors in the P 2 basis {l,x, x 2 }in the P 4 basis 
{1, x, x 2 , x 3 , x 4 }. Compute 

T(l) = x 2 
T (x) = 3x + x 3 
T (x 2 ) = 6 x 2 + x 4 

The coordinates of the input vectors we know are [1, 0, 0] T , [0,1,0] T , and 
[0, 0,1] T . For the output vectors the coordinates are [0, 0,1, 0, 0] T , [0, 3, 0,1, 0] T , 
and [0, 0, 6 , 0,1] T . So, with respect to these two bases, the matrix of the 
transformation is 


A = 


0 0 0 ' 
0 3 0 
1 0 6 
0 10 
0 0 1 


Observe that the dimensionality corresponds with the dimentionality of the 
respective spaces. 

Example 2. 7.5. Let V = R 2 , with S 0 = {vi,v 2 } = {[o],[}]}, Si = 
{wi,w 2 } = {[ 2 ], [ ^l 2 ]}, and T = I the identity. The vectors above are 
expressed in the standard E = {ei,e 2 }, Tvj = Ivj = Vj. To find [vj]s l we 
solve 


Vj = a,jW 1 + ( 3 jW 2 
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v \: 








Therefore 



\ solve linear 
systems 


Si [-f]s 0 



change of 

basis 

matrix 


If 




Note the necessity of using the standard basis to express the vectors in both 
bases and Si. 


2.8 Change of Basis 

Let V be a vector space with bases Sq = {v\... v n } and Si = {rci... w n }, 
and suppose T: V —> V is a linear transformation. We want to find the 
representation of T as a matrix that takes a vector x given in terms of its So 
coordinates and produces the vector Tx given in terms of its Si coordinates. 
We know that x —> [a;]s 0 is well defined. The action of T is known if the 

- Cl -| 

n vector [cc] 5 0 = : and the vectors Tvi,Tv 2 , ■ ■ ■ ,Tv n are known, for if 

- C-n - 

x = EcjVj, then Tx = EcjTvj, by linearity. 

To determine [Tx]s 1 we need to convert the Tvj, j — 1,... ,n to coor¬ 
dinates in the other S\ basis, This is done as follows. Find 
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Then if x 6 V 


[Txjsi = \ZcjTvj\s! = EcjlTvjjs, 



C-+. 1 

c-+- 

5 " 

_ 1 


Cl 

tnl tnn_ 


Cn 


This n x n array [t{j] depends on T, So and Si but not on x. We define the 
S 0 —> Si basis representation of T to be [Uj], and we write this as 


Si 


[T]s 0 = 


til ••• tin 
tnl •■• tnn 


In the special case that T is the identity operator the matrix [7],s 0 converts 
the coordinates of a vector in the basis So to coordinates in the basis Si. It 
is easy to see that 5 0 [I]si must be the inverse of s'J/Jsq and thus 


S 0 [ J ]si • Si [I] So = L 


We can also establish the equality 


SiPIsj = Si [-f] So So [ r ] So So [-H Si- 

In this way we see that the matrix representation of T depends on the bases 
involved. If X is any invertible matrix in M n (F) we can write 

B = X^AX. 


The interpretation in this context is clear 

X: change of coordinate from one basis to another So —» Si 
X : change of coordinate 5i —» So 

A: matrix of the linear transformation in the basis So 
B: matrix of the same linear transformation in the basis Si. 

With this in mind it seems prudent to study linear transformations in the 
basis that makes their matrix representation as simple as possible. 
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Example 2.8.1. Let A 


1 3 

-1 1 


be the matrix representation of a lin¬ 


ear transformation given with respect to the standard basis So = {e\, } = 

{(1, 0), (0,1)} Find the matrix representation of this transformation with 
resepect to the basis Si = {vi, ^ 2 } = {(2,1), (1,1)}. 

Solution. According to the analysis above we need to determine s 0 [7] s and 
S L [I] 5o . Of course 5 1 [I]g Q = g 0 [J]^ 1 . Since the coordinates of the vectors 
in Si are expressed in terms of the basis vectors So we obtain directly 


So i I ]s 1 


2 1 
1 1 


Its inverse is given by 


So [I\si ~ 

Assembling these matrices we have the final matrix converted to the new 
basis. 


1 -1 

-1 2 


Si [A] Sl 


Si [!] s 0 A So Msi 


1 3 


'21' 

-1 1 


1 1 


6 4 

-7 -4 


Example 2.8.2. Consider the same problem as above except that the ma¬ 
trix A is given in the basis Si. Find matrix representation of this transfor¬ 
mation with resepect to the basis S 0 . 

Solution. To solve this problem we need to determine g 0 [A] Sq = g 0 [J] 5i A g 1 [/] s 
As we already have these matrices, we determine that 


So Wsi A Si [I] s 0 ■ 


2 1 

1 1 

1 3 

-1 1 

'1 -1 ' 

-1 2 

r-s 13 1 



—4 8 


So [ A \ So 
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2.9 Appendix A — Solving linear systems 

The key to solving linear systems is to reduce the augmented system to 
RREF and solve the resulting equations. While this may be so, there is an 
intermediate step that occurs about half way through the computation of 
the RREF where the reduced matrix achieves an upper triangular form. At 
this point the solution can be determined directly by back substitution. To 
clarify the rules on back substitution, suppose that we have the triangular 
form 

an ai2 ■ ■ ■ a\ n b\ 

0 022 • • • 0,2n b2 

0 • • • 0 CLnn bn 

Assuming that the diagonal part consists of all nonzero terms, we can solve 
this system by back substitution. First solve for x n = -r 22 -. Now inductively 

“nn 

solve for the remaining solution coordinates using the formula 

j -1 

^ ^ &n—j,n—k%n—k > j 1 , 2 , . . . , Tl 1 

k =0 

This inconvenient looking formula can be replaced by 
1 n 

Xj = — ^ ajkXk ’ j = n- 1, n-.2,..., 1 

L k = j +1 

where the index runs from j = n — 1 up to j m 1. The upshot is that the 
row reduction process can be halted when a triangular-like form has been 
attained. The applies as well to nonsingular and non square systems, where 
the the process is stopped when all the leading ones have been identified, 
entries below them have been zeroed out, and all the zero rows are present. 
The principle reason for using back substitution is to reduce the number 
of computations required, an important consideration in numerical linear 
algebra. In the example below we solve a 3 x 3 nonsingular system. 

Example 2.9.1. Solve Ax = b where 

'12 0 1 [ 3 

A= 2 2-1 b= 6 

-1 3 2 J [ -2 
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Solution. Find the RREF of [A | 6 ], Then solve Ax = b. 


"12 0 

3 

— 2R\ + R 2 

"12 0 

3 ' 

2 2-1 

6 

-> 

0 -2 -1 

0 

-13 2 

-2 

Ri + R 3 

0 5 2 

1 


-\R 2 


-> 


' 1 

2 

0 

3 ' 

0 

1 

1 

2 

0 

0 

5 

2 

1 


— 5i?2 + R 3 
—> 


3 

1 

—2 


— TjRs + i?2 


1 

0 


0 0 1 


12 0 

CO 

0 1 i 

0 

Lo 0 -i 

1 _ 


-2 R 3 




' 1 

2 

0 

3 

0 

1 

1 

2 

0 

0 

0 

1 

-2 


— 2i?2 + Rl 
-> 


' 1 

0 

0 

1 

0 

1 

0 

1 

0 

0 

1 

-2 


Hence solving we obtain *3 = —2, X 2 = 1, and x\ = 1. This is fine, 
but there is a faster way to solve this system. Stop the reduction when 
the system attains a triangular form at (*). From this point solve to obtain 
X 3 = —2. Now back substitute *3 = 2 into the second row (equation) to 
solve for * 2 - Thus X 2 = — ^ (—2) = 1. Finally, back substitute *3 = 2 and 
*2 = 1 into the first row (equation) to solve for x\. Thus X\ = 3 — 2 (1) = 1. 
Sometimes the form (*) is called the row reduced form. 


Example 2.9.2. Given the augmented system for Ax = b is in RREF. 


' 1 

2 

0 

0 

0 

0 


0 

0 

1 

2 

0 

-1 

1 

0 

0 

0 

0 

1 

3 

1 

0 

0 

0 

0 

0 

0 

0 _ 
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Find the solution. 

Solution. The leading ones occur in columns 1, 3, and 5. The values in 
columns 2, 4, and 6 can be taken as free parameters. So, take X2 = r, X4 = s, 
and xq = t. Now solving for the other varables we have 

x\ = 4 — 2 r 
X3 = 1 — 2 s + t 

X 5 = 1 — 3 1 


The solution set is comprised of the vector 
x = [4 — 2r, r, 1 — 2s + t, s, 1 — t,t] T 

= [4, 0,1, 0,1, Of + r [-2,1,0,0,0, of + s [0, 0, -2,1,0, of + t [0, 0,1, 0 - 3, if 
for all r, s, and t. We can rewrite this as the set 



' 4 ' 


' -2 ' 


' 0 ' 


0 




0 


1 


0 


0 




1 


0 


-2 


1 


r, s, t E R or C 


0 

+ r 

0 

+ s 

1 

+ t 

0 



1 


0 


0 


-3 




_ 0 _ 


0 


0 


1 


> 


This representation shows better the connection between the free constants 
and the component vectors that make up the solution. Note this expression 
also reveals the solution of the homogeneous solution Ax = 0 as the set 



2,1,0,0,0, Of + s [0, 0, -2,1, 0, Of + t [0, 0,1,0 


3,1] J 


r,s,t E R or 


Indeed, this is a full subspace. 

Example 2.9.3. The RREF can be used to determine the inverse, as well. 
Given the matrix A E M n , the inverse is given by the matrix X for 
which AX = I. In turn with x\, .. ., x n representing the columns of 
X and ei, ..., e n representing the standard vectors we see that Axj = 
ej. j = 1,2,... ,n. To solve for these vectors, form the augmented matrix 
[A | ej\, j = 1,2,, n and row reduce as above. A massive short cut to 
this process is to augment all the standard vectors at one and row reduce the 
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resulting n x 2n matrix [A \ I], If A is invertible, its RREF is the identity. 
Therefore, 


row 

[A\I\ -> [I | X] 

operations 


and, of course, A 1 = X. Thus, for 


A = 

we row reduce [A \ I] as follows 

[A 1 1] = 


-2 10 
1 1 2 
3 -2 -1 


-2 1 

0 

1 

0 

0 " 



1 1 

2 

0 

1 

0 



3 -2 - 

-1 

0 

0 

1 _ 



row 

1 

0 

0 

3 

; l 

2 

—* 

0 

1 

0 

7 

2 

4 

operations 

0 

0 

1 

- 

5 -1 

-3 


2.10 Exercises 

1. Consider the differential operator T = 2x— (•)—4 acting on the vector 

ax 

space of cubic polynomials, P 3 . Show that T is a linear transformation 
and find a matrix representation of it. Assume the basis is given by 
{1, x, x 2 , x 3 }. 

2. (i) Find matrices A and B, each with positive rank, for which r(A + 
B ) = r(A) + r(B). (ii) Find matrices A and B, each with positive 
rank, for which r(A + B) = 0. (iii) Give a method to find two nonzero 
matrices A and B for which the sum has any preassigned rank. Of 
course, the matrix sizes may depend on this value. 

3. Find square matrices A and B for which r(A) = r(B) = 2 and for 
which r{AB) = 0. 

4. Suppose that A is an m x n matrix and that x is a solution of Ax = b 
over the prescribed field. Show that every solution of Ax = b have the 
form x + xq, where xq is a solution of Ax q = 0 . 
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5. 


6 . 

7. 


9. 


10 . 

11 . 

12 . 

13. 

14. 

15. 

16. 


Find a matrix A £ M n of rank n— 1 for which r(A k ) = n — k for k < n. 
Is it possible to begin this process with a matrix A £ M n of rank n 
and for which r(A k ) = n — k + 1 for k < n ? 

Show that Ax = b has a solution if and only if y T b = 0 if and only if 
y T A = 0 for some column vector. 

In i ?2 the linear transformation that rotates any vector by 6 radians 
counter clockwise T has matrix representation with respect to the 
standard basis given by 


A 


cos 0 — sin 8 

sin 6 cos 6 


What is the matrix representation with respect to the standard basis 
of the transformation that rotates any vector by 0 radians clockwise? 
What is the relation between the matrices? 


Show that if B,C £ M n (F), where B is symmetric and C is skew- 
symmetric, then B = C implies that B = C = 0. 


Prove the general formula for the inverse of the 2x2 matrix A = 


a 

c 


b 

d 


is A 1 = 


1 


det A 

Prove Theorem 2.5.1(a). 


d -b 
—c a 


Prove Theorem 2.5.1(b). 

Prove that every permutation a must have an inverse cr” 1 (i.e. er -1 (<r(j)) 
j ), and the signs of a and a are the same. 


Show that the sign of every transposition is — 1. 
Prove that det A = ^ (-1 ) S9n{a) a v(i) i 
Prove Proposition 2.5.2 using minors. 


Suppose that the n x n matrix A is singular. Show that each column 
of the adjugate matrix A is a solution of Ax = 0. (McDuffee, Chapter 
3, Theorem 29.) 
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17. Suppose that A is an (n-l)xn matrix, and consider the homogeneous 
system Ax = 0 for x € R n . Define hi to be the determinant of the 
(n — 1 ) x (n — 1) matrix formed by removing the i th column of A. Show 
that the vector h = (hi, ... , h n ) T is a solution to Ax = 0. (McDuffee, 
Chapter 3, Corollary 29.) 

18. Show that A = [ _ 1 1 ~ 1 1 ] has no inverse by trying to solve AB = I That 
is, assume the form 


multiply the matrices (A and B) together, and then solve for the un¬ 
knowns a, b, c, and d. (This is not a very efficient way to determine 
inverses of matrices. Try the same thing for any 3x3 matrix.) 

19. Prove that the elementary equation operations do not change the so¬ 
lution set of a linear system. 

20. Find the inverses of E\, E 2 , and E 3 . 

21. Find the matrix representation of linear transformation T that rotates 
any vector by 6 radians counter clockwise (ccw) with respect to the 
basis S = {(2,1), ( 1 ,1)}. 

22. Consider R 3 . Suppose that we have angles {0 l }j =l and pairs of co¬ 
ordinate vectors {(ei, , (ei, e ^), (e 2 , e 3 )}. Let T be the linear trans¬ 

formation that successively rotates a vector in the respective planes 
{(ej 1 ,ej 2 )}* =1 through the respective angles Find the matrix 

representation of T with respect to the standard basis. Prove that it 
is invertible. 

23. Prove Theorem 2.2.4. 

24. Prove or disprove the equivalence of the linear systems. 

2x — 3y = — 1 — x + ly = 3 

x + ly = 5 x + 2y = 3 

25. Find basis for the orthocomplement of the subspace of R 3 spanned by 
the vectors {[ 2 , 1 , 1 ] T , [ 1 , 1 , 2 ] r }. 

26. Find basis for the orthocomplement of the subspace of R 3 spanned by 
the vector [ 1 , 1 , 1 ] T . 
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27. Consider planar rotations in R n with respect to the standard bases 

discounting 


Tl \Tl — 1 ) 

elements. Prove that there must be- - of them 


the particular angle. Display the general representation of any of 
them. Prove or disprove that any two of them are commutative. That 
is for two angles {0j}? =1 and pairs of coordinate vectors {(e^, e* 2 )}^ =1 
the respective counter clockwise rotations are commutative. 


28. Suppose that A € M m k, B e Mk n and both have rank k. Show that 
the rank of AB is k. 

29. Suppose that A € M m k has rank k. Prove that Arref = ^ 

where If. is the identity matrix of size k and 0 is the m — k x k zero 
matrix. 


30. Suppose that B e Mk n has rank k. Prove that Brref = [ h 0 ] 
where Ik is the identity matrix of size k and 0 is tlie/c xn — k zero 
matrix. 


31. Determine and prove a version of Corollary 2.6.1 for 3 x 3 blocked 
matrices, where we assume the diagonal blocks An is invertible and 
wish to conclude the result that the rank of A is the equal to the rank 
of A u . 

32. Suppose that we have angles {0i}f =1 and pairs of coordinate vectors 
{( e ®i j e *a) }^=i - Let T be the linear transformation that successively 
rotates a vector in the respective planes {(eq,ej 2 )}^ =1 through the 
respective angles {#*}f =1 . Prove that the matrix representation of the 
linear transformation with respect to any basis must be invertible. 

33. The super-diagonal of a matrix is the set of elements a^i + \. The subdi¬ 
agonal of a matrix is the set of elements A tri-banded matrix is 

one for which the entries are zero above the super-diagonal and below 
the subdiagonal. Suppose that for an n x n tri-banded matrix T, we 
have cii-ij = a, an = 0, and a^+i = c. Prove the following facts: 


(a) If n is odd det A = 0. 

(b) If n = 2m is even detT = (—l) m a m c m . 

34. For the banded matrix of the previous example, prove the following 
for the powers T p of T. 
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(a) If p is odd, prove that (T v ) i - = 0 if % + j is even. 

(b) If p is even, prove that ( T p )- = 0 if i+j is odd. 

35. Consider the vector space P 2 (1,2) with inner product defined by (p, q) = 
fi p(x) q (a;) dx. Find an orthogonal basis of P 2 (1, 2). (Hint. Begin 
with the standard basis {l,x,x 2 }. Apply the Gram-Schmidt procedure.) 

36. For what values of a and b is the matrix below singular 


A = 


a 2 1 

2 1 b 
1 a —2 


37. The Vandermonde matrix, defined for a sequence of numbers {x\,... x n }, 
is given by the n x 71 matrix 



' 1 

Xl 

4 • 

■■ x"- 1 

Rn = 

1 

X2 

x\ ■ 

■■ 4-' 


1 

X n 

~2 . 

~n— 1 


Prove that the determinant is given by 

n 

det V n = (. Xi-Xj ) 

i>j =1 


38. In the case the x-values are the integers {1 ,,n}, prove that det V n 

n 

is divisible by (i — 1)!. (These numbers are called superfactorials.) 
i =1 

39. Prove that for the weighted functional defined in Remark 2.4.1, it is 
necessary and sufficient that the weights be strictly positive for it to 
be an inner product. 

40. For what values of a and b is the matrix below singular 


A 


b 0 a 0 
0 0 6a 
0 a 0 6 
6 a 0 0 
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41. Find an orthogonal basis for i ?2 from the vectors {(1, 2), (2,1)}. 

42. Find an orthogonal basis of the subspace of R% spanned by {(1, 0,1), (0,1, — 1)} 

43. Suppose that V is a vector space with an inner product, and S C V. 

Show that if S' is a basis of V, S ± = {0}. 

44. Suppose that V is a vector space with an inner product, and S C V. 

Show that if U = 6(S), then U 1 = S' 1 . 

45. Let A e M mn (F). Show it may not be true that r (A) = r(A T A) = 
r(AA T ) unl ess F = R, in which case it is true. 

46. If A £ M n (C) is orthogonal, show that the rows and columns of A are 
orthogonal. 


47. If A is orthogonal then A m is orthogonal for every positive integer m. 
(This is a part of Theorem 2.4.3(b).) 

48. Consider the polynomial space P n [—1,1] with the inner product (p, q) = 
f\P (t) Q (t) dt. Show that every polynomial p G P n for which p (1) = 
p (— 1 ) = 0 is orthogonal to its derivative. 


49. Consider the polynomial space P n [—1,1] with the inner product (p, q) = 
f\ P (t) Q (t) dt. Show that the subspace of polynomials in even pow¬ 
ers (e.g. p (t) = t 2 — 5t 6 ) is orthogonal to the subspace of polynomials 
in odd powers. 


50. Let A = 


1 3 

-1 1 

formation given wit 


be the matrix representation of a linear trans- 


h respect to the standard basis So = {ei, 62 } = 
{(1,0), (0,1)} Find the matrix representation of this transformation 
with resepect to the basis Si = {^ 1 ,^ 2 } = {(2, —3), (1, —2)}. 


51. Show that the sign of every transposition from the set {1,2, ..., n} is 

- 1 . 


52. What are the signs of the permutations {7, 6 , 5, 4, 3, 2, 1} and 
{7,1, 6 , 4, 3, 5, 2} of the integers {1, 2, 3, 4, 5, 6 , 7}? 

53. Prove that the sign of the permutation {m, m — 1,..., 2, 1} is (—l) m . 

54. Suppose A, B e M n (F). If A is singular, use a row space argument to 
show that det AB = 0. 
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55. Show that if A e M mjn and B e M mjn and both r(A) = r(B) = m. If 
r(AB) «= m — k, what can be said about n? 

56. Prove that 


det 


Xl 

X2 

X 3 

X4 

~X2 

Xl 

—X4 

X3 


X4 

Xl 

-x 2 

— X 4 

-X3 

~X2 

Xl 


2\2 


= (xf + x\ + x\ + xf) 


57. Prove that there is no invertible 3x3 matrix that has all the same 
cofactors. What similar statement can be made for n x n matrices? 

58. Show by example that there are matrices A and B for which lin^^oo A n 
and lim )WOO B n both exist, but for which lim n _ > . 0O ( AB) n does not ex¬ 
ist. 

59. Let A € M 2 (C). Show that there is no matrix solution B € M 2 (C) 
to AB — BA = I. What can you say about the same problem with 
A, Be M n (C)? 

60. Show by example that if AC = BC then it does not follow that A = B. 
However, show that if C is inveritble the conclusion A = B is valid. 



